Skip to content

fix: lex a template substitution inline instead of pre-scanning it - #52

Merged
marevol merged 1 commit into
es6/fix-regex-after-lookaheadfrom
es6/fix-template-substitution
Aug 28, 2026
Merged

fix: lex a template substitution inline instead of pre-scanning it#52
marevol merged 1 commit into
es6/fix-regex-after-lookaheadfrom
es6/fix-template-substitution

Conversation

@marevol

@marevol marevol commented Aug 28, 2026

Copy link
Copy Markdown
Contributor

Stacked on #51.

Any slash inside a substitution broke the template, division included:

var a = 6, b = 3;
print(`${a / b}`);   // Expected an operand but found template_tail
print(`${1 / 2}`);   // Expected an operand but found template_tail
print(`${/c+/.source}`);               // Expected an operand but found /
print(`${"aa".replace(/a/g, "b")}`);   // Expected an operand but found /

Cause

scanTemplate pre-scanned the substitution by counting braces, then handed the region to a nested lexer and called lexify() on it exactly once. But lexify() deliberately returns as soon as it emits an ambiguous token so the parser can decide whether a / starts a regular expression — so one call stopped at the first slash and the rest of the expression was never tokenized. The parser then met TEMPLATE_TAIL where it wanted an operand.

Looping lexify() would have fixed division and not the regular expression: the literal is only recoverable while its token is the last one in the stream.

Fix

The substitution is no longer pre-scanned. scanTemplate now emits just the part that opens the template and returns; the ordinary lexer loop tokenizes the expression, and the brace matching the opening ${ brings it back to emit TEMPLATE_MIDDLE or TEMPLATE_TAIL. Which brace that is comes from a small stack of brace depths, one entry per open substitution, so templates still nest.

That stack is lexer state, so the speculative lookahead added in #51 has to rewind it along with the position. Leaving it behind puts the lexer back inside a substitution while the depth says otherwise, and the brace that ends the substitution is then read as an ordinary one.

Things the brace pre-scan also got wrong

It knew about strings but not about comments or regular expressions:

print(`x${ 1 // }
}y`);                        // printed "x1\n}y", now "x1y" — silently wrong before
print(`x${ /* } */ 1 }y`);   // was a SyntaxError
print(`x${ 1 /* don't */ }y`);  // the apostrophe ate the rest of the file
print(`${ /}/.source }`);    // the brace in the regex ended the substitution

skipStringInSubstitution() existed only for that pre-scan and is removed.

One separate off-by-one goes with it: the part scanner consumed an escaped line terminator without counting the line, because it used else if (isEOL(ch0)) where scanString uses a nested if. Every line number after such a template was one too low. A template and a string with the same escaped newline now report the same line.

Verification

./gradlew build testOptimistic testPessimistic

suite before after
test 665, 0 fail 665, 0 fail
testOptimistic 1715, 0 fail 1715, 0 fail
testPessimistic 1715, 0 fail 1715, 0 fail

Any slash inside a substitution broke the template, division included:

    var a = 6, b = 3;
    print(`${a / b}`);            Expected an operand but found template_tail
    print(`${1 / 2}`);            Expected an operand but found template_tail
    print(`${/c+/.source}`);      Expected an operand but found /
    print(`${"aa".replace(/a/g, "b")}`);   Expected an operand but found /

scanTemplate pre-scanned the substitution by counting braces, then handed the
region to a nested lexer and called lexify() on it exactly once. But lexify()
deliberately returns as soon as it emits an ambiguous token so the parser can
decide whether a "/" starts a regular expression - so one call stopped at the
first slash and the rest of the expression was never tokenized. The parser then
met TEMPLATE_TAIL where it wanted an operand.

Looping lexify() would have fixed division and not the regular expression: the
literal is only recoverable while its token is the last one in the stream.

So the substitution is no longer pre-scanned. scanTemplate now emits just the
part that opens the template and returns; the ordinary lexer loop tokenizes the
expression, and the brace matching the opening ${ brings it back to emit
TEMPLATE_MIDDLE or TEMPLATE_TAIL. Which brace that is comes from a small stack
of brace depths, one entry per open substitution, so templates still nest.

That stack is lexer state, so the speculative lookahead added for arrow
functions has to rewind it along with the position. Leaving it behind puts the
lexer back inside a substitution while the depth says otherwise, and the brace
that ends the substitution is then read as an ordinary one.

Letting the real loop do the work fixes three more things the brace pre-scan got
wrong, because it knew about strings but not about comments or regular
expressions:

    print(`x${ 1 // }
    }y`);                     printed "x1\n}y", now "x1y" - silently wrong before
    print(`x${ /* } */ 1 }y`);          was a SyntaxError
    print(`x${ 1 /* don't */ }y`);      the apostrophe ate the rest of the file
    print(`${ /}/.source }`);           the brace in the regex ended the substitution

skipStringInSubstitution() existed only for that pre-scan and is removed.

One separate off-by-one goes with it: the part scanner consumed an escaped line
terminator without counting the line, because it used "else if (isEOL(ch0))"
where scanString uses a nested if. Every line number after such a template was
one too low. A template and a string with the same escaped newline now report
the same line.

./gradlew build testOptimistic testPessimistic:

  suite            before         after
  test             665, 0 fail    665, 0 fail
  testOptimistic   1715, 0 fail   1715, 0 fail
  testPessimistic  1715, 0 fail   1715, 0 fail
@marevol
marevol force-pushed the es6/fix-template-substitution branch from 4067b64 to bb0c0ea Compare August 28, 2026 05:11
@marevol marevol self-assigned this Aug 28, 2026
@marevol
marevol merged commit 61279fd into master Aug 28, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant